<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.9.1"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>AIfES 2: Tutorial training F32</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
</script>
<script type="text/javascript" async="async" src="https://cdn.jsdelivr.net/npm/mathjax@2/MathJax.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="AIfES_logo_small.png"/></td>
  <td id="projectalign" style="padding-left: 0.5em;">
   <div id="projectname">AIfES 2
   &#160;<span id="projectnumber">2.0.0</span>
   </div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.9.1 -->
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
var searchBox = new SearchBox("searchBox", "search",false,'Search','.html');
/* @license-end */
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
$(function() {
  initMenu('',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
/* @license-end */</script>
<div id="main-nav"></div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
  <div id="nav-tree">
    <div id="nav-tree-contents">
      <div id="nav-sync" class="sync"></div>
    </div>
  </div>
  <div id="splitbar" style="-moz-user-select:none;" 
       class="ui-resizable-handle">
  </div>
</div>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&amp;dn=gpl-2.0.txt GPL-v2 */
$(document).ready(function(){initNavTree('_tutorial_training_f32.html',''); initResizable(); });
/* @license-end */
</script>
<div id="doc-content">
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div class="PageDoc"><div class="header">
  <div class="headertitle">
<div class="title">Tutorial training F32 </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p><a class="anchor" id="md__builds_ims_esa_aifes_2_doc_md_04_tutorial_training"></a> This tutorial should explain how the different components of AIfES 2 work together to train a simple Feed-Forward Neural Network (FNN) or Multi Layer Perceptron (MLP). If you just want to perform an inference with pretrained weights, switch to the <a class="el" href="_tutorial_inference_f32.html">inference tutorial</a>.</p>
<h1><a class="anchor" id="autotoc_md16"></a>
Example</h1>
<p>As an example, we take a robot with two powered wheels and a RGB color sensor that should follow a black line on a white paper. To fulfill the task, we map the color sensor values with an FNN directly to the control commands for the two wheel-motors of the robot. The inputs for the FNN are the RGB color values scaled to the interval [0, 1] and the outputs are either "on" (1) or "off" (0) to control the motors.</p>
<div class="image">
<img src="example_line_follow_robot.png" alt="" width="600px"/>
</div>
<p>The following cases should be considered:</p><ol type="1">
<li>The sensor points to a black area (RGB = [0, 0, 0]): The robot is too far on the left and should turn on the left wheel-motor while removing power from the right motor.</li>
<li>The sensor points to a white area (RGB = [1, 1, 1]): The robot is too far on the right and should turn on the right wheel-motor while removing power from the left motor.</li>
<li>The sensor points to a red area (RGB = [1, 0, 0]): The robot reached the stop mark and should switch off both motors.</li>
</ol>
<p>The resulting input data of the FNN is then</p>
<table class="markdownTable">
<tr class="markdownTableHead">
<th class="markdownTableHeadNone">R   </th><th class="markdownTableHeadNone">G   </th><th class="markdownTableHeadNone">B    </th></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">0    </td></tr>
<tr class="markdownTableRowEven">
<td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">1    </td></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">0   </td></tr>
</table>
<p>and the output should be</p>
<table class="markdownTable">
<tr class="markdownTableHead">
<th class="markdownTableHeadNone">left motor   </th><th class="markdownTableHeadNone">right motor    </th></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">1   </td><td class="markdownTableBodyNone">0    </td></tr>
<tr class="markdownTableRowEven">
<td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">1    </td></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">0   </td><td class="markdownTableBodyNone">0   </td></tr>
</table>
<h1><a class="anchor" id="autotoc_md17"></a>
Design the neural network</h1>
<p>To set up the FNN in AIfES 2, we need to design the structure of the neural network. It needs three inputs for the RGB color values and two outputs for the two motors. Because the task is rather easy, we use just one hidden layer with three neurons.</p>
<div class="image">
<img src="example_line_follow_abstract_model.png" alt="" width="800px"/>
</div>
<p>To create the network in AIfES, it must be divided into logical layers like the fully-connected (dense) layer and the activation functions. We choose a Leaky ReLU activation for the hidden layer and a Sigmoid activation for the output layer.</p>
<div class="image">
<img src="example_line_follow_model.png" alt="" width="800px"/>
</div>
<h1><a class="anchor" id="autotoc_md18"></a>
Create the neural network in AIfES</h1>
<p>AIfES provides implementations of the layers for different data types that can be optimized for several hardware platforms. An overview of the layers that are available for training can be seen in the <a class="el" href="index.html#OverviewTraining">overview section of the main page</a>. In the overview table you can click on the layer in the first column for a description on how the layer works. To see how to create the layer in code, choose one of the implementations for your used data type and click on the link. <br  />
 In this tutorial we work with the float 32 data type (<a class="el" href="aimath__f32_8h.html">F32 </a>) and use the default implementations (without any hardware specific optimizations) of the layers.</p>
<p>Used layer types:</p><ul>
<li><a class="el" href="ailayer__input_8h.html">Input layer </a></li>
<li><a class="el" href="ailayer__dense_8h.html">Dense layer </a></li>
<li><a class="el" href="ailayer__leaky__relu_8h.html">Leaky ReLU layer </a></li>
<li><a class="el" href="ailayer__sigmoid_8h.html">Sigmoid layer </a></li>
</ul>
<p>Used implementations:</p><ul>
<li><a class="el" href="ailayer__input__default_8h.html#af7bb369ff05cf9cca9fda5765a951c79" title="Initializes and connect an Input layer  with the F32  default implementation.">ailayer_input_f32_default()</a></li>
<li><a class="el" href="ailayer__dense__default_8h.html#a5f58ad071502178879dccb0acab8b74b" title="Initializes and connect a Dense layer  with the F32  default implementation.">ailayer_dense_f32_default()</a></li>
<li><a class="el" href="ailayer__leaky__relu__default_8h.html#aecb15e9008d56b41a8370d9f38cde60c" title="Initializes and connect a Leaky ReLU layer  with the F32  default implementation.">ailayer_leaky_relu_f32_default()</a></li>
<li><a class="el" href="ailayer__sigmoid__default_8h.html#a1ec45b121e81b85e2109f0072d46f602" title="Initializes and connect a Sigmoid layer  with the F32  default implementation.">ailayer_sigmoid_f32_default()</a></li>
</ul>
<h2><a class="anchor" id="autotoc_md19"></a>
Declaration and configuration of the layers</h2>
<p>For every layer we need to create a variable of the specific layer type and configure it for our needs. See the documentation of the data type and hardware specific implementations (for example <a class="el" href="ailayer__dense__default_8h.html#a5f58ad071502178879dccb0acab8b74b" title="Initializes and connect a Dense layer  with the F32  default implementation.">ailayer_dense_f32_default()</a>) for code examples on how to configure the layers.</p>
<p>Our designed network can be declared with the following code </p><div class="fragment"><div class="line"><span class="comment">// The main model structure that holds the whole neural network</span></div>
<div class="line"><a class="code" href="structaimodel.html">aimodel_t</a> model;</div>
<div class="line"> </div>
<div class="line"> </div>
<div class="line"><span class="comment">// The layer structures for F32 data type and their configurations</span></div>
<div class="line">uint16_t input_layer_shape[] = {1, 3};</div>
<div class="line"><a class="code" href="structailayer__input.html">ailayer_input_f32_t</a> input_layer           = AILAYER_INPUT_F32_A(2, input_layer_shape);</div>
<div class="line"><a class="code" href="structailayer__dense.html">ailayer_dense_f32_t</a> dense_layer_1         = AILAYER_DENSE_F32_A(3);</div>
<div class="line"><a class="code" href="structailayer__leaky__relu__f32.html">ailayer_leaky_relu_f32_t</a> leaky_relu_layer = AILAYER_LEAKY_RELU_F32_A(0.01f);</div>
<div class="line"><a class="code" href="structailayer__dense.html">ailayer_dense_f32_t</a> dense_layer_2         = AILAYER_DENSE_F32_A(2);</div>
<div class="line"><a class="code" href="structailayer__sigmoid.html">ailayer_sigmoid_f32_t</a> sigmoid_layer       = AILAYER_SIGMOID_F32_A();</div>
<div class="ttc" id="astructailayer__dense_html"><div class="ttname"><a href="structailayer__dense.html">ailayer_dense</a></div><div class="ttdoc">General Dense layer  structure.</div><div class="ttdef"><b>Definition:</b> ailayer_dense.h:71</div></div>
<div class="ttc" id="astructailayer__input_html"><div class="ttname"><a href="structailayer__input.html">ailayer_input</a></div><div class="ttdoc">General Input layer  structure.</div><div class="ttdef"><b>Definition:</b> ailayer_input.h:39</div></div>
<div class="ttc" id="astructailayer__leaky__relu__f32_html"><div class="ttname"><a href="structailayer__leaky__relu__f32.html">ailayer_leaky_relu_f32</a></div><div class="ttdoc">Data-type specific Leaky ReLU layer struct for F32 .</div><div class="ttdef"><b>Definition:</b> ailayer_leaky_relu_default.h:51</div></div>
<div class="ttc" id="astructailayer__sigmoid_html"><div class="ttname"><a href="structailayer__sigmoid.html">ailayer_sigmoid</a></div><div class="ttdoc">General Sigmoid layer  struct.</div><div class="ttdef"><b>Definition:</b> ailayer_sigmoid.h:47</div></div>
<div class="ttc" id="astructaimodel_html"><div class="ttname"><a href="structaimodel.html">aimodel</a></div><div class="ttdoc">AIfES artificial neural network model.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:181</div></div>
</div><!-- fragment --><p>We use the initializer macros with the "_A" at the end, because we call <a class="el" href="aialgo__sequential__inference_8h.html#a26db68cb4231b534b03c649d6eeab3f8" title="Assign the memory for the trainable parameters (like weights, bias, ...) of the model.">aialgo_distribute_parameter_memory()</a> later on to set the remaining paramenters (like the weights) automatically.</p>
<h2><a class="anchor" id="autotoc_md20"></a>
Connection and initialization of the layers</h2>
<p>Afterwards the layers are connected and initialized with the data type and hardware specific implementations </p><div class="fragment"><div class="line"><span class="comment">// Layer pointer to perform the connection</span></div>
<div class="line"><a class="code" href="structailayer.html">ailayer_t</a> *x;</div>
<div class="line"> </div>
<div class="line">model.<a class="code" href="structaimodel.html#a708a94e69112ad215b2b52da2238a711">input_layer</a> = <a class="code" href="ailayer__input__default_8h.html#af7bb369ff05cf9cca9fda5765a951c79">ailayer_input_f32_default</a>(&amp;input_layer);</div>
<div class="line">x = <a class="code" href="ailayer__dense__default_8h.html#a5f58ad071502178879dccb0acab8b74b">ailayer_dense_f32_default</a>(&amp;dense_layer_1, model.<a class="code" href="structaimodel.html#a708a94e69112ad215b2b52da2238a711">input_layer</a>);</div>
<div class="line">x = <a class="code" href="ailayer__leaky__relu__default_8h.html#aecb15e9008d56b41a8370d9f38cde60c">ailayer_leaky_relu_f32_default</a>(&amp;leaky_relu_layer, x);</div>
<div class="line">x = <a class="code" href="ailayer__dense__default_8h.html#a5f58ad071502178879dccb0acab8b74b">ailayer_dense_f32_default</a>(&amp;dense_layer_2, x);</div>
<div class="line">x = <a class="code" href="ailayer__sigmoid__default_8h.html#a1ec45b121e81b85e2109f0072d46f602">ailayer_sigmoid_f32_default</a>(&amp;sigmoid_layer, x);</div>
<div class="line">model.<a class="code" href="structaimodel.html#a7c7ad89e7d15631b3f5893b8f19030ef">output_layer</a> = x;</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Finish the model creation by checking the connections and setting some parameters for further processing</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a3fb665166082f1e7a89e23218a105ce8">aialgo_compile_model</a>(&amp;model);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a3fb665166082f1e7a89e23218a105ce8"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a3fb665166082f1e7a89e23218a105ce8">aialgo_compile_model</a></div><div class="ttdeci">uint8_t aialgo_compile_model(aimodel_t *model)</div><div class="ttdoc">Initialize the model structure.</div></div>
<div class="ttc" id="aailayer__dense__default_8h_html_a5f58ad071502178879dccb0acab8b74b"><div class="ttname"><a href="ailayer__dense__default_8h.html#a5f58ad071502178879dccb0acab8b74b">ailayer_dense_f32_default</a></div><div class="ttdeci">ailayer_t * ailayer_dense_f32_default(ailayer_dense_f32_t *layer, ailayer_t *input_layer)</div><div class="ttdoc">Initializes and connect a Dense layer  with the F32  default implementation.</div></div>
<div class="ttc" id="aailayer__input__default_8h_html_af7bb369ff05cf9cca9fda5765a951c79"><div class="ttname"><a href="ailayer__input__default_8h.html#af7bb369ff05cf9cca9fda5765a951c79">ailayer_input_f32_default</a></div><div class="ttdeci">ailayer_t * ailayer_input_f32_default(ailayer_input_f32_t *layer)</div><div class="ttdoc">Initializes and connect an Input layer  with the F32  default implementation.</div></div>
<div class="ttc" id="aailayer__leaky__relu__default_8h_html_aecb15e9008d56b41a8370d9f38cde60c"><div class="ttname"><a href="ailayer__leaky__relu__default_8h.html#aecb15e9008d56b41a8370d9f38cde60c">ailayer_leaky_relu_f32_default</a></div><div class="ttdeci">ailayer_t * ailayer_leaky_relu_f32_default(ailayer_leaky_relu_f32_t *layer, ailayer_t *input_layer)</div><div class="ttdoc">Initializes and connect a Leaky ReLU layer  with the F32  default implementation.</div></div>
<div class="ttc" id="aailayer__sigmoid__default_8h_html_a1ec45b121e81b85e2109f0072d46f602"><div class="ttname"><a href="ailayer__sigmoid__default_8h.html#a1ec45b121e81b85e2109f0072d46f602">ailayer_sigmoid_f32_default</a></div><div class="ttdeci">ailayer_t * ailayer_sigmoid_f32_default(ailayer_sigmoid_f32_t *layer, ailayer_t *input_layer)</div><div class="ttdoc">Initializes and connect a Sigmoid layer  with the F32  default implementation.</div></div>
<div class="ttc" id="astructailayer_html"><div class="ttname"><a href="structailayer.html">ailayer</a></div><div class="ttdoc">AIfES layer interface.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:252</div></div>
<div class="ttc" id="astructaimodel_html_a708a94e69112ad215b2b52da2238a711"><div class="ttname"><a href="structaimodel.html#a708a94e69112ad215b2b52da2238a711">aimodel::input_layer</a></div><div class="ttdeci">ailayer_t * input_layer</div><div class="ttdoc">Input layer of the model that gets the input data.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:182</div></div>
<div class="ttc" id="astructaimodel_html_a7c7ad89e7d15631b3f5893b8f19030ef"><div class="ttname"><a href="structaimodel.html#a7c7ad89e7d15631b3f5893b8f19030ef">aimodel::output_layer</a></div><div class="ttdeci">ailayer_t * output_layer</div><div class="ttdoc">Output layer of the model.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:183</div></div>
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md21"></a>
Set the memory for the trainable parameters</h2>
<p>Because AIfES doesn't allocate any memory on its own, you have to set the memory buffers for the trainable parameters like the weights and biases manually. Therefore you can choose fully flexible, where the parameters should be located in memory. To calculate the required amount of memory by the model, the <a class="el" href="aialgo__sequential__inference_8h.html#acdf3763b8fe9047446ddbcfdbdae5570" title="Calculate the memory requirements for the trainable parameters (like weights, bias,...">aialgo_sizeof_parameter_memory()</a> function can be used or the size can be calculated manually (See the sizeof_paramem() functions of every layer for the required amount of memory and the set_paramem() function for how to set it to the layer). <br  />
 With <a class="el" href="aialgo__sequential__inference_8h.html#a26db68cb4231b534b03c649d6eeab3f8" title="Assign the memory for the trainable parameters (like weights, bias, ...) of the model.">aialgo_distribute_parameter_memory()</a> a memory block of the required size can be distributed and set to the different trainable parameters of the model.</p>
<p>A dynamic allocation of the memory using malloc could look like the following: </p><div class="fragment"><div class="line">uint32_t parameter_memory_size = <a class="code" href="aialgo__sequential__inference_8h.html#acdf3763b8fe9047446ddbcfdbdae5570">aialgo_sizeof_parameter_memory</a>(&amp;model);</div>
<div class="line"><span class="keywordtype">void</span> *parameter_memory = malloc(parameter_memory_size);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Distribute the memory to the trainable parameters of the model</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a26db68cb4231b534b03c649d6eeab3f8">aialgo_distribute_parameter_memory</a>(&amp;model, parameter_memory, parameter_memory_size);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a26db68cb4231b534b03c649d6eeab3f8"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a26db68cb4231b534b03c649d6eeab3f8">aialgo_distribute_parameter_memory</a></div><div class="ttdeci">void aialgo_distribute_parameter_memory(aimodel_t *model, void *memory_ptr, uint32_t memory_size)</div><div class="ttdoc">Assign the memory for the trainable parameters (like weights, bias, ...) of the model.</div></div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_acdf3763b8fe9047446ddbcfdbdae5570"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#acdf3763b8fe9047446ddbcfdbdae5570">aialgo_sizeof_parameter_memory</a></div><div class="ttdeci">uint32_t aialgo_sizeof_parameter_memory(aimodel_t *model)</div><div class="ttdoc">Calculate the memory requirements for the trainable parameters (like weights, bias,...</div></div>
</div><!-- fragment --><p>You could also pre-define a memory buffer if you know the size in advance, for example </p><div class="fragment"><div class="line"><span class="keyword">const</span> uint32_t parameter_memory_size = 80;</div>
<div class="line"><span class="keywordtype">char</span> parameter_memory[parameter_memory_size];</div>
<div class="line"> </div>
<div class="line">...</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Distribute the memory to the trainable parameters of the model</span></div>
<div class="line">aialgo_distribute_parameter_memory(&amp;model, parameter_memory, parameter_memory_size);</div>
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md22"></a>
Print the layer structure to the console</h2>
<p>To see the structure of your created model, you can print a model summary to the console </p><div class="fragment"><div class="line">aiprint(<span class="stringliteral">&quot;\n-------------- Model structure ---------------\n&quot;</span>);</div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a3bb5fdb556c51ad8b3043d220f0a1276">aialgo_print_model_structure</a>(&amp;model);</div>
<div class="line">aiprint(<span class="stringliteral">&quot;----------------------------------------------\n\n&quot;</span>);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a3bb5fdb556c51ad8b3043d220f0a1276"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a3bb5fdb556c51ad8b3043d220f0a1276">aialgo_print_model_structure</a></div><div class="ttdeci">void aialgo_print_model_structure(aimodel_t *model)</div><div class="ttdoc">Print the layer structure of the model with the configured parameters.</div></div>
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md23"></a>
Train the neural network</h1>
<p>The created FNN can now be trained. AIfES provides the necessary functions to train the model with the backpropagation algorithm.</p>
<h2><a class="anchor" id="autotoc_md24"></a>
Configure the loss</h2>
<p>To calculate the gradients for the backpropagation training algorithm, a loss function must be configured to the model. An overview of available losses can be seen in the <a class="el" href="index.html#OverviewTraining">overview section of the main page</a>. In the overview table you can click on the loss in the first column for a description on how the loss works. To see how to create the loss in code, choose one of the implementations for your used data type and click on the link.</p>
<p>In this tutorial we work with the float 32 data type (<a class="el" href="aimath__f32_8h.html">F32 </a>) and use the default implementation (without any hardware specific optimizations) of the loss. We choose the Cross-Entropy loss because we have sigmoid outputs of the network and binary target data. (Cross-Entropy loss will not work for other output layers than sigmoid or softmax!)</p>
<p>Used loss type:</p><ul>
<li><a class="el" href="ailoss__crossentropy_8h.html">Cross-Entropy loss </a></li>
</ul>
<p>Used implementation:</p><ul>
<li><a class="el" href="ailoss__crossentropy__default_8h.html#aa10b5c43deef1891d98cb8c45d57b3ee" title="Initializes and connect a Cross-Entropy loss  with the F32  default implementation using a mean reduc...">ailoss_crossentropy_f32_default()</a></li>
</ul>
<p>The loss can be declared, connected and initialized similar to the layers, by calling</p>
<div class="fragment"><div class="line"><a class="code" href="structailoss__crossentropy.html">ailoss_crossentropy_f32_t</a> crossentropy_loss;</div>
<div class="line"> </div>
<div class="line">model.<a class="code" href="structaimodel.html#ad08c61cef46d4042c62d8cdeba81986f">loss</a> = <a class="code" href="ailoss__crossentropy__default_8h.html#aa10b5c43deef1891d98cb8c45d57b3ee">ailoss_crossentropy_f32_default</a>(&amp;crossentropy_loss, model.<a class="code" href="structaimodel.html#a7c7ad89e7d15631b3f5893b8f19030ef">output_layer</a>);</div>
<div class="ttc" id="aailoss__crossentropy__default_8h_html_aa10b5c43deef1891d98cb8c45d57b3ee"><div class="ttname"><a href="ailoss__crossentropy__default_8h.html#aa10b5c43deef1891d98cb8c45d57b3ee">ailoss_crossentropy_f32_default</a></div><div class="ttdeci">ailoss_t * ailoss_crossentropy_f32_default(ailoss_crossentropy_f32_t *loss, ailayer_t *input_layer)</div><div class="ttdoc">Initializes and connect a Cross-Entropy loss  with the F32  default implementation using a mean reduc...</div></div>
<div class="ttc" id="astructailoss__crossentropy_html"><div class="ttname"><a href="structailoss__crossentropy.html">ailoss_crossentropy</a></div><div class="ttdoc">General Cross-Entropy loss  struct.</div><div class="ttdef"><b>Definition:</b> ailoss_crossentropy.h:62</div></div>
<div class="ttc" id="astructaimodel_html_ad08c61cef46d4042c62d8cdeba81986f"><div class="ttname"><a href="structaimodel.html#ad08c61cef46d4042c62d8cdeba81986f">aimodel::loss</a></div><div class="ttdeci">ailoss_t * loss</div><div class="ttdoc">The loss or cost function of the model (only for training).</div><div class="ttdef"><b>Definition:</b> aifes_core.h:188</div></div>
</div><!-- fragment --><p>You can print the loss configuration to the console for debugging purposes with </p><div class="fragment"><div class="line"><a class="code" href="aialgo__sequential__training_8h.html#ab19a3d00e7ac130806780cc90e317a09">aialgo_print_loss_specs</a>(model.<a class="code" href="structaimodel.html#ad08c61cef46d4042c62d8cdeba81986f">loss</a>);</div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_ab19a3d00e7ac130806780cc90e317a09"><div class="ttname"><a href="aialgo__sequential__training_8h.html#ab19a3d00e7ac130806780cc90e317a09">aialgo_print_loss_specs</a></div><div class="ttdeci">void aialgo_print_loss_specs(ailoss_t *loss)</div><div class="ttdoc">Print the loss specs.</div></div>
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md25"></a>
Configure the optimizer</h2>
<p>To update the parameters with the calculated gradients, an optimizer must be created. An overview of available optimizers can be seen in the <a class="el" href="index.html#OverviewTraining">overview section of the main page</a>. In the overview table you can click on the optimizer in the first column for a description on how the loss works. To see how to create the optimizer in code, choose one of the implementations for your used data type and click on the link.</p>
<p>In this tutorial we work with the float 32 data type (<a class="el" href="aimath__f32_8h.html">F32 </a>) and use the default implementation (without any hardware specific optimizations) of the optimizer. We choose the Adam optimizer that converges really fast to the optimum with just a small memory and computation overhead. On more limited systems, the SGD optimizer might be a better choice.</p>
<p>Used optimizer type:</p><ul>
<li><a class="el" href="aiopti__adam_8h.html">Adam optimizer </a></li>
</ul>
<p>Used implementation:</p><ul>
<li><a class="el" href="aiopti__adam__default_8h.html#a403603bdc0d77e2d1cbc3fd4cd37880a" title="Initializes an Adam optimizer  with the F32  default implementation.">aiopti_adam_f32_default()</a></li>
</ul>
<p>The optimizer can be declared and configured by calling</p>
<div class="fragment"><div class="line"><a class="code" href="structaiopti__adam__f32.html">aiopti_adam_f32_t</a> adam_opti = {</div>
<div class="line">    .<a class="code" href="structaiopti__adam__f32.html#a7fa6b6264ed55a45f7af8b00eae05f59">learning_rate</a> = 0.01f,</div>
<div class="line"> </div>
<div class="line">    .beta1 = 0.9f,</div>
<div class="line">    .beta2 = 0.999f,</div>
<div class="line">    .eps = 1e-7f</div>
<div class="line">};</div>
<div class="ttc" id="astructaiopti__adam__f32_html"><div class="ttname"><a href="structaiopti__adam__f32.html">aiopti_adam_f32</a></div><div class="ttdoc">Data-type specific Adam optimizer  struct for F32 .</div><div class="ttdef"><b>Definition:</b> aiopti_adam_default.h:45</div></div>
<div class="ttc" id="astructaiopti__adam__f32_html_a7fa6b6264ed55a45f7af8b00eae05f59"><div class="ttname"><a href="structaiopti__adam__f32.html#a7fa6b6264ed55a45f7af8b00eae05f59">aiopti_adam_f32::learning_rate</a></div><div class="ttdeci">aiscalar_f32_t learning_rate</div><div class="ttdoc">Storage for aiopti.learning_rate scalar in F32.</div><div class="ttdef"><b>Definition:</b> aiopti_adam_default.h:54</div></div>
</div><!-- fragment --><p>in <b>C</b> or</p>
<div class="fragment"><div class="line"><a class="code" href="structaiopti__adam__f32.html">aiopti_adam_f32_t</a> adam_opti = AIOPTI_ADAM_F32(0.01f, 0.9f, 0.999f, 1e-7f);</div>
</div><!-- fragment --><p>in <b>C++</b> and on Arduino. Afterwards it can be initialized by</p>
<div class="fragment"><div class="line"><a class="code" href="structaiopti.html">aiopti_t</a> *optimizer = <a class="code" href="aiopti__adam__default_8h.html#a403603bdc0d77e2d1cbc3fd4cd37880a">aiopti_adam_f32_default</a>(&amp;adam_opti);</div>
<div class="ttc" id="aaiopti__adam__default_8h_html_a403603bdc0d77e2d1cbc3fd4cd37880a"><div class="ttname"><a href="aiopti__adam__default_8h.html#a403603bdc0d77e2d1cbc3fd4cd37880a">aiopti_adam_f32_default</a></div><div class="ttdeci">aiopti_t * aiopti_adam_f32_default(aiopti_adam_f32_t *opti)</div><div class="ttdoc">Initializes an Adam optimizer  with the F32  default implementation.</div></div>
<div class="ttc" id="astructaiopti_html"><div class="ttname"><a href="structaiopti.html">aiopti</a></div><div class="ttdoc">AIfES optimizer interface.</div><div class="ttdef"><b>Definition:</b> aifes_core.h:438</div></div>
</div><!-- fragment --><p>You can print the optimizer configuration to the console for debugging purposes with </p><div class="fragment"><div class="line"><a class="code" href="aialgo__sequential__training_8h.html#aa0a7ab4189d2f68c675d42aa687758e6">aialgo_print_optimizer_specs</a>(optimizer);</div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_aa0a7ab4189d2f68c675d42aa687758e6"><div class="ttname"><a href="aialgo__sequential__training_8h.html#aa0a7ab4189d2f68c675d42aa687758e6">aialgo_print_optimizer_specs</a></div><div class="ttdeci">void aialgo_print_optimizer_specs(aiopti_t *opti)</div><div class="ttdoc">Print the optimizer specs.</div></div>
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md26"></a>
Initialize the trainable parameters</h2>
<p>The trainable parameters like the weights and biases have to be initialized before the training. There are different initialization techniques for different models. For a simple FNN, the recommended initialization of the weights and biases of the dense layers are dependent on the connected activation functions</p>
<table class="markdownTable">
<tr class="markdownTableHead">
<th class="markdownTableHeadNone">Activation function   </th><th class="markdownTableHeadNone">Weights-init   </th><th class="markdownTableHeadNone">Bias-init    </th></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">None, tanh, sigmoid, softmax   </td><td class="markdownTableBodyNone">Glorot   </td><td class="markdownTableBodyNone">Zeros    </td></tr>
<tr class="markdownTableRowEven">
<td class="markdownTableBodyNone">ReLU and variants   </td><td class="markdownTableBodyNone">He   </td><td class="markdownTableBodyNone">Zeros    </td></tr>
<tr class="markdownTableRowOdd">
<td class="markdownTableBodyNone">SELU   </td><td class="markdownTableBodyNone">LeCun   </td><td class="markdownTableBodyNone">Zeros   </td></tr>
</table>
<p>In our example, the two dense layers can be initialized manually like described in the recommendation table (uniform distribution versions) </p><div class="fragment"><div class="line"><span class="comment">// Set the seed for your configured random function for example with</span></div>
<div class="line">srand(time(0));</div>
<div class="line"> </div>
<div class="line"><a class="code" href="aimath__f32__default_8h.html#a11d31daf4d011186446fd77a253b9f7f">aimath_f32_default_init_he_uniform</a>(&amp;dense_layer_1.<a class="code" href="structailayer__dense.html#abaaf545fb7c9d3a0b80aa5a460202e04">weights</a>);</div>
<div class="line"><a class="code" href="aimath__f32__default_8h.html#a760163e6cee0f2a5e341069ad20bfcde">aimath_f32_default_init_zeros</a>(&amp;dense_layer_1.<a class="code" href="structailayer__dense.html#a87664225b3b1ad7ec4cb74cc8bd8f349">bias</a>);</div>
<div class="line"> </div>
<div class="line"><a class="code" href="aimath__f32__default_8h.html#aae8d8c59a70df08142d376e134944317">aimath_f32_default_init_glorot_uniform</a>(&amp;dense_layer_2.<a class="code" href="structailayer__dense.html#abaaf545fb7c9d3a0b80aa5a460202e04">weights</a>);</div>
<div class="line"><a class="code" href="aimath__f32__default_8h.html#a760163e6cee0f2a5e341069ad20bfcde">aimath_f32_default_init_zeros</a>(&amp;dense_layer_2.<a class="code" href="structailayer__dense.html#a87664225b3b1ad7ec4cb74cc8bd8f349">bias</a>);</div>
<div class="ttc" id="aaimath__f32__default_8h_html_a11d31daf4d011186446fd77a253b9f7f"><div class="ttname"><a href="aimath__f32__default_8h.html#a11d31daf4d011186446fd77a253b9f7f">aimath_f32_default_init_he_uniform</a></div><div class="ttdeci">void aimath_f32_default_init_he_uniform(aitensor_t *tensor)</div><div class="ttdoc">Fills a F32  tensor with uniformly drawn random numbers within given range, according to He et al.</div></div>
<div class="ttc" id="aaimath__f32__default_8h_html_a760163e6cee0f2a5e341069ad20bfcde"><div class="ttname"><a href="aimath__f32__default_8h.html#a760163e6cee0f2a5e341069ad20bfcde">aimath_f32_default_init_zeros</a></div><div class="ttdeci">void aimath_f32_default_init_zeros(aitensor_t *tensor)</div><div class="ttdoc">Fills a F32  tensor with zeros.</div></div>
<div class="ttc" id="aaimath__f32__default_8h_html_aae8d8c59a70df08142d376e134944317"><div class="ttname"><a href="aimath__f32__default_8h.html#aae8d8c59a70df08142d376e134944317">aimath_f32_default_init_glorot_uniform</a></div><div class="ttdeci">void aimath_f32_default_init_glorot_uniform(aitensor_t *tensor)</div><div class="ttdoc">Fills a F32  tensor with random numbers uniformly within given range, according to Glorot et al.</div></div>
<div class="ttc" id="astructailayer__dense_html_a87664225b3b1ad7ec4cb74cc8bd8f349"><div class="ttname"><a href="structailayer__dense.html#a87664225b3b1ad7ec4cb74cc8bd8f349">ailayer_dense::bias</a></div><div class="ttdeci">aitensor_t bias</div><div class="ttdoc">Tensor containing the layer bias weights.</div><div class="ttdef"><b>Definition:</b> ailayer_dense.h:88</div></div>
<div class="ttc" id="astructailayer__dense_html_abaaf545fb7c9d3a0b80aa5a460202e04"><div class="ttname"><a href="structailayer__dense.html#abaaf545fb7c9d3a0b80aa5a460202e04">ailayer_dense::weights</a></div><div class="ttdeci">aitensor_t weights</div><div class="ttdoc">Tensor containing the layer weights.</div><div class="ttdef"><b>Definition:</b> ailayer_dense.h:87</div></div>
</div><!-- fragment --><p>AIfES can also automatically initialize the trainable parameters of the whole model with the default initialization methods configured to the layers. </p><div class="fragment"><div class="line"><span class="comment">// Set the seed for your configured random function for example with</span></div>
<div class="line">srand(time(0));</div>
<div class="line"><a class="code" href="aialgo__sequential__training_8h.html#ab6ea378b18812cf68fb4ae70c57380a5">aialgo_initialize_parameters_model</a>(&amp;model);</div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_ab6ea378b18812cf68fb4ae70c57380a5"><div class="ttname"><a href="aialgo__sequential__training_8h.html#ab6ea378b18812cf68fb4ae70c57380a5">aialgo_initialize_parameters_model</a></div><div class="ttdeci">void aialgo_initialize_parameters_model(aimodel_t *model)</div><div class="ttdoc">Initialize the parameters of the given model with their default initialization method.</div></div>
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md27"></a>
Allocate and initialize the working memory</h2>
<p>Because AIfES doesn't allocate any memory on its own, you have to set the memory buffers for the training manually. This memory is required for example for the intermediate results of the layers, for the gradients and as working memory for the optimizer (for momentums). Therefore you can choose fully flexible, where the buffer should be located in memory. To calculate the required amount of memory for the training, the <a class="el" href="aialgo__sequential__training_8h.html#aaa72bf9da57a600c0d4fef4ba03f725e" title="Calculate the memory requirements for model training.">aialgo_sizeof_training_memory()</a> function can be used. <br  />
 With <a class="el" href="aialgo__sequential__training_8h.html#aa6ae098c2add3651d216724f102e931b" title="Assign the memory for model training.">aialgo_schedule_training_memory()</a> a memory block of the required size can be distributed and scheduled (memory regions might be shared over time) to the model.</p>
<p>A dynamic allocation of the memory using malloc could look like the following: </p><div class="fragment"><div class="line">uint32_t training_memory_size = <a class="code" href="aialgo__sequential__training_8h.html#aaa72bf9da57a600c0d4fef4ba03f725e">aialgo_sizeof_training_memory</a>(&amp;model, optimizer);</div>
<div class="line"><span class="keywordtype">void</span> *training_memory = malloc(training_memory_size);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Schedule the memory to the model</span></div>
<div class="line"><a class="code" href="aialgo__sequential__training_8h.html#aa6ae098c2add3651d216724f102e931b">aialgo_schedule_training_memory</a>(&amp;model, optimizer, training_memory, training_memory_size);</div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_aa6ae098c2add3651d216724f102e931b"><div class="ttname"><a href="aialgo__sequential__training_8h.html#aa6ae098c2add3651d216724f102e931b">aialgo_schedule_training_memory</a></div><div class="ttdeci">uint8_t aialgo_schedule_training_memory(aimodel_t *model, aiopti_t *optimizer, void *memory_ptr, uint32_t memory_size)</div><div class="ttdoc">Assign the memory for model training.</div></div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_aaa72bf9da57a600c0d4fef4ba03f725e"><div class="ttname"><a href="aialgo__sequential__training_8h.html#aaa72bf9da57a600c0d4fef4ba03f725e">aialgo_sizeof_training_memory</a></div><div class="ttdeci">uint32_t aialgo_sizeof_training_memory(aimodel_t *model, aiopti_t *optimizer)</div><div class="ttdoc">Calculate the memory requirements for model training.</div></div>
</div><!-- fragment --><p>You could also pre-define a memory buffer if you know the size in advance, for example </p><div class="fragment"><div class="line"><span class="keyword">const</span> uint32_t training_memory_size = 772;</div>
<div class="line"><span class="keywordtype">char</span> training_memory[training_memory_size];</div>
<div class="line"> </div>
<div class="line">...</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Schedule the memory to the model</span></div>
<div class="line">aialgo_schedule_training_memory(&amp;model, optimizer, training_memory, training_memory_size);</div>
</div><!-- fragment --><p>Before starting the training, the scheduled memory must be initialized. The initialization sets for example the momentums of the optimizer to zero.</p>
<div class="fragment"><div class="line"><a class="code" href="aialgo__sequential__training_8h.html#a62a55277765b0bdd5c88955eb92e4c13">aialgo_init_model_for_training</a>(&amp;model, optimizer);</div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_a62a55277765b0bdd5c88955eb92e4c13"><div class="ttname"><a href="aialgo__sequential__training_8h.html#a62a55277765b0bdd5c88955eb92e4c13">aialgo_init_model_for_training</a></div><div class="ttdeci">void aialgo_init_model_for_training(aimodel_t *model, aiopti_t *optimizer)</div><div class="ttdoc">Initialize the optimization memory of the model layers.</div></div>
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md28"></a>
Perform the training</h2>
<p>To perform the training, the input data and the target data / labels (and also the test data if available) must be packed in a tensor to be processed by AIfES. A tensor in AIfES is just a N-dimensional array that is used to hold the data values in a structured way. To do this, create 2D tensors of the used data type. The shape describes the size of the dimensions of the tensor. The first dimension (the rows) is the batch dimension, i.e. the dimension of the different training samples. The second dimension equals the inputs / outputs of the neural network.</p>
<div class="fragment"><div class="line">uint16_t x_train_shape[2] = {3, 3};</div>
<div class="line"><span class="keywordtype">float</span> x_train_data[3*3] = {0.0f, 0.0f, 0.0f,</div>
<div class="line">                           1.0f, 1.0f, 1.0f,</div>
<div class="line">                           1.0f, 0.0f, 0.0f};</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> x_train = AITENSOR_2D_F32(x_train_shape, x_train_data);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Target data / Labels for training</span></div>
<div class="line">uint16_t y_train_shape[2] = {3, 2};</div>
<div class="line"><span class="keywordtype">float</span> y_train_data[3*2] = {1.0f, 0.0f,</div>
<div class="line">                           0.0f, 1.0f,</div>
<div class="line">                           0.0f, 0.0f};</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> y_train = AITENSOR_2D_F32(y_train_shape, y_train_data);</div>
<div class="ttc" id="astructaitensor_html"><div class="ttname"><a href="structaitensor.html">aitensor</a></div><div class="ttdoc">A tensor in AIfES.</div><div class="ttdef"><b>Definition:</b> aifes_math.h:89</div></div>
</div><!-- fragment --><p>Because in this case we are just interested in the outputs of the FNN on our training dataset, our test data is just the training data. </p><div class="fragment"><div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> *x_test = &amp;x_train;</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> *y_test = &amp;y_train;</div>
</div><!-- fragment --><p>Now everything is ready to perform the actual training. AIfES provides the function <a class="el" href="aialgo__sequential__training_8h.html#a6557fccf302f653bcdcb77830463d14d" title="Perform one training epoch on all data batches of the dataset using backpropagation.">aialgo_train_model()</a> that performs one epoch of training with mini-batches. One epoch means that the model has seen the whole data ones afterwards. The function <a class="el" href="aialgo__sequential__training_8h.html#ab06a58b69f3136374e2dc6664f56e0c7" title="Calculate the loss in F32  data type.">aialgo_calc_loss_model_f32()</a> can be used to calculate the loss on the test data between the epochs.</p>
<p>An example training with these functions that trains the model for a pre-defined amount of epochs might look like this: </p><div class="fragment"><div class="line"><span class="keywordtype">int</span> epochs = 100;</div>
<div class="line"><span class="keywordtype">int</span> batch_size = 3;</div>
<div class="line"><span class="keywordtype">int</span> print_interval = 10;</div>
<div class="line"> </div>
<div class="line"><span class="keywordtype">float</span> loss;</div>
<div class="line">aiprint(<span class="stringliteral">&quot;\nStart training\n&quot;</span>);</div>
<div class="line"><span class="keywordflow">for</span>(<span class="keywordtype">int</span> i = 0; i &lt; epochs; i++)</div>
<div class="line">{</div>
<div class="line">    <span class="comment">// One epoch of training. Iterates through the whole data once</span></div>
<div class="line">    <a class="code" href="aialgo__sequential__training_8h.html#a6557fccf302f653bcdcb77830463d14d">aialgo_train_model</a>(&amp;model, &amp;x_train, &amp;y_train, optimizer, batch_size);</div>
<div class="line"> </div>
<div class="line">    <span class="comment">// Calculate and print loss every print_interval epochs</span></div>
<div class="line">    <span class="keywordflow">if</span>(i % print_interval == 0)</div>
<div class="line">    {</div>
<div class="line">        <a class="code" href="aialgo__sequential__training_8h.html#ab06a58b69f3136374e2dc6664f56e0c7">aialgo_calc_loss_model_f32</a>(&amp;model, x_test, y_test, &amp;loss);</div>
<div class="line">        </div>
<div class="line">        <span class="comment">// Print the loss to the console</span></div>
<div class="line">        aiprint(<span class="stringliteral">&quot;Epoch &quot;</span>); aiprint_int(<span class="stringliteral">&quot;%5d&quot;</span>, i);</div>
<div class="line">        aiprint(<span class="stringliteral">&quot;: test loss: &quot;</span>); aiprint_float(<span class="stringliteral">&quot;%f&quot;</span>, loss);</div>
<div class="line">        aiprint(<span class="stringliteral">&quot;\n&quot;</span>);</div>
<div class="line">    }</div>
<div class="line">}</div>
<div class="line">aiprint(<span class="stringliteral">&quot;Finished training\n\n&quot;</span>);</div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_a6557fccf302f653bcdcb77830463d14d"><div class="ttname"><a href="aialgo__sequential__training_8h.html#a6557fccf302f653bcdcb77830463d14d">aialgo_train_model</a></div><div class="ttdeci">uint8_t aialgo_train_model(aimodel_t *model, aitensor_t *input_tensor, aitensor_t *target_tensor, aiopti_t *optimizer, uint32_t batch_size)</div><div class="ttdoc">Perform one training epoch on all data batches of the dataset using backpropagation.</div></div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_ab06a58b69f3136374e2dc6664f56e0c7"><div class="ttname"><a href="aialgo__sequential__training_8h.html#ab06a58b69f3136374e2dc6664f56e0c7">aialgo_calc_loss_model_f32</a></div><div class="ttdeci">uint8_t aialgo_calc_loss_model_f32(aimodel_t *model, aitensor_t *input_data, aitensor_t *target_data, float *result)</div><div class="ttdoc">Calculate the loss in F32  data type.</div></div>
</div><!-- fragment --><p>If you like to have more control over the training process, you can also use the more advanced functions</p><ul>
<li><a class="el" href="aialgo__sequential__training_8h.html#a20ba9e0bdcfd4e36bc1168789de7e99f" title="Set the gradients to zero.">aialgo_zero_gradients_model()</a></li>
<li><a class="el" href="aialgo__sequential__inference_8h.html#a4655caab3051cc837312c286fbe4789a" title="Perform a forward pass on the model.">aialgo_forward_model()</a></li>
<li><a class="el" href="aialgo__sequential__training_8h.html#aca4ec290b2db30cc76ad78aff649a69d" title="Perform the backward pass.">aialgo_backward_model()</a></li>
<li><a class="el" href="aialgo__sequential__training_8h.html#a4650ca244c2d5086eb80190d2416f86e" title="Perform the optimization step on the model parameters.">aialgo_update_params_model()</a></li>
</ul>
<p>The call order of the functions in pseudo code could look like this</p>
<div class="fragment"><div class="line"><span class="keywordflow">for</span> epochs</div>
<div class="line">    <span class="keywordflow">for each</span> batch in dataset</div>
<div class="line">        <a class="code" href="aialgo__sequential__training_8h.html#a20ba9e0bdcfd4e36bc1168789de7e99f">aialgo_zero_gradients_model</a>()</div>
<div class="line">        for each sample in the batch</div>
<div class="line">            <a class="code" href="aialgo__sequential__inference_8h.html#a4655caab3051cc837312c286fbe4789a">aialgo_forward_model</a>()</div>
<div class="line">            <a class="code" href="aialgo__sequential__training_8h.html#aca4ec290b2db30cc76ad78aff649a69d">aialgo_backward_model</a>()</div>
<div class="line">        endfor</div>
<div class="line">        <a class="code" href="aialgo__sequential__training_8h.html#a4650ca244c2d5086eb80190d2416f86e">aialgo_update_params_model</a>()</div>
<div class="line">    endfor</div>
<div class="line">endfor</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a4655caab3051cc837312c286fbe4789a"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a4655caab3051cc837312c286fbe4789a">aialgo_forward_model</a></div><div class="ttdeci">aitensor_t * aialgo_forward_model(aimodel_t *model, aitensor_t *input_data)</div><div class="ttdoc">Perform a forward pass on the model.</div></div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_a20ba9e0bdcfd4e36bc1168789de7e99f"><div class="ttname"><a href="aialgo__sequential__training_8h.html#a20ba9e0bdcfd4e36bc1168789de7e99f">aialgo_zero_gradients_model</a></div><div class="ttdeci">void aialgo_zero_gradients_model(aimodel_t *model, aiopti_t *optimizer)</div><div class="ttdoc">Set the gradients to zero.</div></div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_a4650ca244c2d5086eb80190d2416f86e"><div class="ttname"><a href="aialgo__sequential__training_8h.html#a4650ca244c2d5086eb80190d2416f86e">aialgo_update_params_model</a></div><div class="ttdeci">void aialgo_update_params_model(aimodel_t *model, aiopti_t *optimizer)</div><div class="ttdoc">Perform the optimization step on the model parameters.</div></div>
<div class="ttc" id="aaialgo__sequential__training_8h_html_aca4ec290b2db30cc76ad78aff649a69d"><div class="ttname"><a href="aialgo__sequential__training_8h.html#aca4ec290b2db30cc76ad78aff649a69d">aialgo_backward_model</a></div><div class="ttdeci">void aialgo_backward_model(aimodel_t *model, aitensor_t *target_data)</div><div class="ttdoc">Perform the backward pass.</div></div>
</div><!-- fragment --><p>For a code example, see the source code of the <a class="el" href="aialgo__sequential__training_8h.html#a6557fccf302f653bcdcb77830463d14d" title="Perform one training epoch on all data batches of the dataset using backpropagation.">aialgo_train_model()</a> function.</p>
<h1><a class="anchor" id="autotoc_md29"></a>
Test the trained model</h1>
<p>To test the trained model you can perform an inference on the test data and compare the results with the targets. To do this you can call the inference function <a class="el" href="aialgo__sequential__inference_8h.html#a295cb2bb6c3fefc478042f66e067e090" title="Perform an inference on the model / Run the model.">aialgo_inference_model()</a>.</p>
<div class="fragment"><div class="line"><span class="comment">// Create an empty tensor for the inference results</span></div>
<div class="line">uint16_t y_out_shape[2] = {3, 2};</div>
<div class="line"><span class="keywordtype">float</span> y_out_data[3*2];</div>
<div class="line"><a class="code" href="structaitensor.html">aitensor_t</a> y_out = AITENSOR_2D_F32(y_out_shape, y_out_data);</div>
<div class="line"> </div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a295cb2bb6c3fefc478042f66e067e090">aialgo_inference_model</a>(&amp;model, x_test, &amp;y_out);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a295cb2bb6c3fefc478042f66e067e090"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a295cb2bb6c3fefc478042f66e067e090">aialgo_inference_model</a></div><div class="ttdeci">uint8_t aialgo_inference_model(aimodel_t *model, aitensor_t *input_data, aitensor_t *output_data)</div><div class="ttdoc">Perform an inference on the model / Run the model.</div></div>
</div><!-- fragment --><p>Afterwards you can print the results to the console for further inspection: </p><div class="fragment"><div class="line">aiprint(<span class="stringliteral">&quot;x_test:\n&quot;</span>);</div>
<div class="line"><a class="code" href="aimath__basic_8h.html#ab10c8d06990943806f0be8fcc6af03fc">print_aitensor</a>(x_test);</div>
<div class="line">aiprint(<span class="stringliteral">&quot;NN output:\n&quot;</span>);</div>
<div class="line"><a class="code" href="aimath__basic_8h.html#ab10c8d06990943806f0be8fcc6af03fc">print_aitensor</a>(&amp;y_out);</div>
<div class="ttc" id="aaimath__basic_8h_html_ab10c8d06990943806f0be8fcc6af03fc"><div class="ttname"><a href="aimath__basic_8h.html#ab10c8d06990943806f0be8fcc6af03fc">print_aitensor</a></div><div class="ttdeci">void print_aitensor(const aitensor_t *tensor)</div><div class="ttdoc">Printing a tensor to console.</div></div>
</div><!-- fragment --><h1><a class="anchor" id="autotoc_md30"></a>
Change working memory</h1>
<p>If you are satisfied with your trained model, you can delete the working memory created for the training (if it was created dynamically). For further inferences / usage of the model you should assign a new memory block, because the big training memory is not needed anymore. Use <a class="el" href="aialgo__sequential__inference_8h.html#a7cbfac6a46c02107d19af7c8f6e5469a" title="Assign the memory for intermediate results of an inference to the model.">aialgo_schedule_inference_memory()</a> to assign a (much smaller) memory block to the model that is made for inference. The required size can be calculated with <a class="el" href="aialgo__sequential__inference_8h.html#a877ce6eee19a9f9bcbdb115d83537e68" title="Calculate the memory requirements for intermediate results of an inference.">aialgo_sizeof_inference_memory()</a>.</p>
<p>Example: </p><div class="fragment"><div class="line">uint32_t inference_memory_size = <a class="code" href="aialgo__sequential__inference_8h.html#a877ce6eee19a9f9bcbdb115d83537e68">aialgo_sizeof_inference_memory</a>(&amp;model);</div>
<div class="line"><span class="keywordtype">void</span> *inference_memory = malloc(inference_memory_size);</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Schedule the memory to the model</span></div>
<div class="line"><a class="code" href="aialgo__sequential__inference_8h.html#a7cbfac6a46c02107d19af7c8f6e5469a">aialgo_schedule_inference_memory</a>(&amp;model, inference_memory, inference_memory_size);</div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a7cbfac6a46c02107d19af7c8f6e5469a"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a7cbfac6a46c02107d19af7c8f6e5469a">aialgo_schedule_inference_memory</a></div><div class="ttdeci">uint8_t aialgo_schedule_inference_memory(aimodel_t *model, void *memory_ptr, uint32_t memory_size)</div><div class="ttdoc">Assign the memory for intermediate results of an inference to the model.</div></div>
<div class="ttc" id="aaialgo__sequential__inference_8h_html_a877ce6eee19a9f9bcbdb115d83537e68"><div class="ttname"><a href="aialgo__sequential__inference_8h.html#a877ce6eee19a9f9bcbdb115d83537e68">aialgo_sizeof_inference_memory</a></div><div class="ttdeci">uint32_t aialgo_sizeof_inference_memory(aimodel_t *model)</div><div class="ttdoc">Calculate the memory requirements for intermediate results of an inference.</div></div>
</div><!-- fragment --><p> Or </p><div class="fragment"><div class="line"><span class="keyword">const</span> uint32_t inference_memory_size = 24;</div>
<div class="line"><span class="keywordtype">char</span> inference_memory[inference_memory_size];</div>
<div class="line"> </div>
<div class="line">...</div>
<div class="line"> </div>
<div class="line"><span class="comment">// Schedule the memory to the model</span></div>
<div class="line">aialgo_schedule_inference_memory(&amp;model, inference_memory, inference_memory_size);</div>
</div><!-- fragment --> </div></div><!-- contents -->
</div><!-- PageDoc -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
  <ul>
    <li class="footer">Generated by <a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.9.1 </li>
  </ul>
</div>
</body>
</html>
